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Abstract 


We combine a large multi-site randomized control trial with administrative and survey data 
to demonstrate that intensive advising during high school and college leads to large increases 
in bachelor’s degree attainment. Novel causal forest methods suggest that these increases are 
driven primarily by improvements in the quality of initial enrollment. Program effects are 
consistent across sites, cohorts, advisors, and student characteristics, suggesting the model is 
scalable. While current and proposed investments in postsecondary education focus on cutting 
costs, our result suggest that investment in advising is likely to be a more efficient route to 
promote bachelor’s degree attainment. 
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1 Introduction 


A college education remains an effective channel through which children born into low-income fami- 
lies can achieve greater economic opportunity. Among those born to parents in the bottom quintile, 
youth who attend college are two and a half times as likely to make it to the top income quintile 
(Chetty et al. 2017). The economic and non-pecuniary benefits of college are most pronounced 
among bachelor’s degree-holders. For instance, adults with a BA earn near $20,000 more per year 
than adults with some college credits but no degree (Ma, Pender, and Welch, 2019). 

At the same time, socioeconomic gaps in college completion have widened; while half of people 
from high-income families obtain a bachelor’s degree by age 25, only one in ten people from low- 
income families do (Bailey and Dynarski, 2011). Differences in preparation explain some of the 
attainment gap, but disparities in college success by family income persist even upon control for 
academic achievement (Bailey and Dynarski, 2011; Belley and Lochner, 2007). 

These persistent disparities have motivated efforts to increase degree attainment among stu- 
dents from lower income backgrounds. Most efforts have centered on the role of college costs, 
including significant expansions to the need-based federal Pell Grant and the adoption and contin- 
ued promotion of “free college” programs. While much academic and policy attention has focused 
on affordability barriers as a central factor contributing to socioeconomic gaps in college completion, 
evidence suggests that a lack of financial resources may not be the primary driver of gaps in de- 
gree attainment (Bulman et al. 2017). Increasing research shows that many financial aid packages 
have more modest effects on degree attainment than one might expect given the generosity of the 
awards (Angrist, Autor, and Pallais 2020; Barr et al. 2021; Bettinger et al. 2019; Dynarski 2008; 
Fitzpatrick and Jones 2016; Sjoquist and Winters 2012). One reason for these relatively modest 
impacts may be that most aid packages do little or nothing to address informational and assistance 
barriers that likely impede low-income students from navigating the application process or selecting 
institutions from which they are most likely to graduate and receive a meaningful return. In this 
paper, we investigation how the provision of personalized college guidance and support via intensive 
advising influences bachelor’s degree attainment. 


College advising programs are an important supplement to the limited college advising students 


receive within their high schools.! These initiatives are widespread — college advising organizations 
serve two million students per year (more than half the number of high-school graduates each 


2 Yet despite the volume of programs and the magnitude of financial investment in these 


year). 
organizations, rigorous evidence of the impact of advising on students’ college success is fairly 
limited. While a few studies demonstrate increased enrollment in college, those that track students 
beyond initial enrollment suggest that these effects fade (Avery 2013; Carrell and Sacerdote 2017). 
In Avery (2013), the estimated “intent to treat” effect for any enrollment became negative from 
the fall to spring semester, while the estimated effect on four-year enrollment fell roughly a third, 
from 0.151 to 0.097. In Carrell and Sacerdote (2017), those assigned to treatment were roughly 
4 percentage points less likely to be enrolled in the second year, conditional on enrollment in the 
first.2 Our study is the first to estimate the extent to which intensive college advising generates 
increases in bachelor’s degree receipt. Given the sizeable share of students who drop out from 
college after the first year and the many benefits associated with degree receipt, it is critical to 
understand whether initial enrollment effects translate into increased degree attainment. 

Our paper contributes new, precisely-identified evidence of the effect of an intensive college 
advising program on low-income students’ bachelor’s degree attainment.4 We conducted a multi- 
cohort and multi-site, randomized controlled trial of the Bottom Line (BL) college advising program, 
which operates in several cities in Massachusetts, New York, and Illinois, drawing students from 
several hundred high schools.° 

The BL model is divided into two distinct stages: Access advising and Success advising. Access 
advisors provide individualized advising to students from the summer before senior year of high 
school through the summer after high school. Like many college access programs, advisors work 
with students on identifying well-matched colleges to apply to and on completing and submitting 


college applications. Advisors encourage students to apply to and attend four-year colleges and 


‘College guidance only accounts for approximately 20 percent of the typical high school advisor’s workload (Bridge- 
land and Bruce 2011). 

*Estimates from National College Attainment Network (NCAN). 

3An ongoing evaluation of an intensive advising program like BL seems to suggest that effects persist into the 
third year, but has not yet followed students to degree attainment (Castleman, Deutschlander, and Lohner, 2020). 

“Bottom Line serves students from families that make less than 200 percent of the federal poverty line. 

°Bottom Line is also in the process of exploring scaling to additional states. BL’s site in Chicago opened in 2014 
and is not included in our analysis. 


universities, with the goal of increasing bachelor’s degree attainment among the population of 
economically-disadvantaged, academically college-ready students. In particular advisors encourage 
students to consider a set of target institutions that BL has identified as providing students with 
an optimal combination of quality and affordability. Success advising is a unique component of 
the BL model. For students who enroll at one of the target institutions (approximately 50 percent 
of advisees choose to do so), BL continues to provide individualized, campus-based support to 
students for up to six years following high school. 

We find that students randomly offered BL advising are substantially more likely to earn a 
bachelor’s degree within 5-6 years of high school. Pooling across experimental cohorts (high school 
graduating classes of 2015 and 2016), students offered Bottom Line were 7.6 percentage points 
more likely to earn a bachelor’s degree within five years of high school; this represents a 16 percent 
increase relative to the control group. Students in the first experimental cohort were 9.6 percentage 
points more likely to earn a bachelor’s degree within six years of high school, an 18 percent increase 
relative to the control group. Bottom Line’s impact on degree attainment stems almost entirely from 
treated students graduating from institutions with above median levels of post-enrollment earnings 
and social mobility, and low student loan default rates. These effects on bachelor’s attainment are 
as large as any rigorous estimates we have come across in the economics literature. 

In the face of declining social mobility, intensive advising may be an effective policy strategy to 
promote greater economic opportunity for economically-disadvantaged, academically college-ready 
students. Indeed, the estimated impact of advising on bachelor’s degree attainment is roughly as 
large as the (conditional on aptitude) gap in degree attainment between children from families 
in the first and fourth quartile of the income distribution (Belley and Lochner 2007). While the 
observed degree effects are quite consistent across different types of students, the fact that BL 
primarily serves students of color further suggests that substantial expansion of the BL model 
could contribute to increased racial equity and mobility as well. 

Degree effects appear driven primarily through shifts to higher quality enrollment. Pooling 
across cohorts, students offered BL were 9.1 percentage points (13 percent) more likely to attend 


four-year institutions, with almost all of this marginal enrollment occurring at institutions with 


high graduation rates.® We use novel causal forest methods to demonstrate the strong relationship 
between the likelihood that BL shifts an individual into a higher quality institution and the like- 
lihood that BL increases that individual’s BA attainment.’ Individuals whom BL likely shifts to 
higher quality institutions (those with above-median PTEs on initial enrollment at an institution 
with an above-median graduation rate) appear to account for nearly all of the estimated treatment 
effects on bachelor’s degree attainment. Consistent with this conclusion, conditioning on initial en- 
rollment and enrollment quality in a simple mediation analysis suggests that most of the treatment 
effect on BA attainment operates through initial college choice, with perhaps a modest role for the 
ongoing advising that BL provides to some students while in college.® 

Detailed advisor-student interaction data and student surveys indicate that advisors appear 
to play a particularly important role in shaping school choice. Advisors spent an average of 10-15 
hours with each student from the summer before senior year of high school through when students 
transition to college. While treated and control students were equally likely to apply to college 
and for financial aid, treatment students applied to significantly more colleges. Furthermore, most 
students noted that BL staff were very important in their application and decision process. 

We also leverage the quasi-random assignment of advisors to investigate how advising behavior, 
as well as advisor backgrounds and characteristics, affect student outcomes and to investigate the 
potential scalability of the BL model.? We find little relationship between advisor demographics or 
behaviors and advisor effectiveness. Indeed, when we estimate advisor fixed effects, 29 out of the 30 
advisors have a positive effect on four-year college enrollment.!° This lack of heterogeneity across 


different types of advisors suggests that BL has a well-developed set of advisor recruitment, selec- 


°We estimate an 8.8 percentage point (32 percent) increase in the likelihood that an individual attends a four-year 
college with an above-median graduation rate. 

"To do so we estimate personalized treatment effects (PTEs) on enrollment at an institution with above-median 
graduation rates and PTEs on BA attainment using causal forest methods. Causal forests use machine learning to 
estimate complex relationships between baseline covariates, treatment assignment, and outcomes of interest. Based 
on these relationships and each subject’s value on the covariates, causal forests then estimate individual treatment 
effects for each subject. We provide further explanation of causal forest methods in the Mechanisms section below 
(Athey and Imbens, 2016; Athey and Wager, 2019). 

’The estimated “treatment effect” on BA attainment falls to 1.6 percentage points once we condition on whether 
an individual enrolls in a two-year, four-year, or four-year with above-median graduation rates in the fall after 
high-school graduation. This result is only suggestive, however, given the strong assumptions underlying mediation 
analyses. 

°Student baseline characteristics are balanced across advisors, consistent with random assignment. 

An F-test on the advisor fixed effects indicates that we cannot reject the null of equivalent advisor effects on 
four-year college enrollment (p-value of 0.902). 


tion, training, and development processes for ensuring advisor success, and supports the potential 
scalability of the model, since results do not appear to be driven by a small set of particularly 
high-performing advisors.!! 

Taken collectively, these results suggest that the BL model has the potential to maintain its 
positive impact with diverse populations in numerous settings. To put the magnitude of these 
effects into context, Figure 1 illustrates that the direct cost per additional bachelor’s degree gen- 
erated by intensive advising appears to be substantially lower than that for programs that provide 
significant financial assistance for college. Aid programs increase bachelor’s degree attainment by 
one percentage point or less per $1,000, with most rigorously-evaluated aid programs increasing 
attainment by less than 0.5 percentage points per $1,000. By contrast, BL advising increases bach- 
elor’s degree attainment by over two percentage points per $1,000. Indeed, the advising model 
appears to be as cost-effective at increasing bachelor’s degree attainment as any rigorougly evalu- 
ated policy of which we are aware. While maintaining implementation quality at scale for BL may 
be more challenging than for aid programs, the consistency of BL’s impacts across sites, student 
groups, and advisors suggests that providing cities with a combination of financial and technical 
assistance to replicate the model could be a more efficient approach to increasing bachelor’s degree 


attainment than offering students additional aid. 


2 Background 


Our study builds on a relatively recent body of work aimed at rigorously evaluating the short-term 
effects of college advising. Unlike with financial aid, the structure of how advising is provided has 
afforded few opportunities for quasi-experimental evaluation. This has resulted in a series of het- 
erogenous and small-scale randomized investigations of advising programs. Carrell and Sacerdote 
(2017) find evidence that a combination of near-peer mentoring and fee waivers led to substan- 
tial increases in the share of female students who completed at least two years of college, while 


Avery (2013)’s pilot RCT of a program (N=238) providing standardized test tutoring in addition 


''The New York City program only began with the high-school graduating class of 2012 and thus provides a more 
direct test of the scalability of the program. We find similarly large effects of the program there. 


to assistance with college choice and applications found positive effects on four-year enrollment.!? 


While both studies provide suggestive evidence that advising increases initial enrollment, neither 
study investigates impacts on degree attainment or persistence beyond the second year in college. 
Furthermore, in both cases the estimated effect of advising on enrollment appeared to dissipate over 
time, suggesting that marginal enrollees were less likely to persist. In Avery (2013), the estimated 
“intent to treat” effect for any enrollment became negative from the fall to spring semester, while 
the estimated effect on four-year enrollment fell roughly a third, from 0.151 to 0.097. In Carrell 
and Sacerdote (2017), those assigned to treatment were roughly 4 percentage points less likely to 
be enrolled in the second year, conditional on enrollment in the first. 

Bos et al. (2012) evaluate a similar near-peer mentoring program in a somewhat poorer and 
more Hispanic population in Los Angeles and find no overall effects on enrollment or four-year 
enrollment. Other evaluations of large-scale quasi-advising programs targeted at poor students 
(such as the federally-funded Upward Bound program) found no impact on enrollment (Seftor, 
Mamun, and Shirn, 2009).'% 

An important contrast between Bottom Line and these advising programs is the intensity of 
the program model. As we describe in the next section, Bottom Line employs professional advisors 
and supports students during high school and throughout college (if they attend a BL “target” 
college, which approximately 50% do). As we show in Figure 2, the average student spends 10-15 
hours with their advisor before transitioning to college, which represents a more intensive level of 


engagement than the programs described above. 


2.1 Bottom Line 


BL began in Boston in 1997 and now operates programs in Boston, Worcester (MA), New York 


City, and Chicago. Students are initially admitted into the Access program, which provides students 


™ Avery also has two interesting small-scale studies (Avery 2010; Avery 2014) that focus on very high achieving 
students and the extent to which mentoring (or tele-mentoring) can influence college match, finding suggestive effects 
on college match. 

'3In a somewhat different context (Ontario), Oreopoulos and Ford (2016) evaluate a schoolwide intervention in 
which students participated in a series of school-based workshops in which they received guidance on choosing 
postsecondary programs to pursue, completed applications, and applied for financial aid. Students assigned to these 
workshops were 2.9 percentage points more likely to go to college, but the effect was driven entirely by increasing 
enrollment at community college, and the study does not provide longitudinal data on whether treated students 
persisted in or graduated from college at higher rates. 


with college and financial aid application support during high school. BL actively promotes the 
Access program through high schools and non-profit partners in each community. Students apply 
to the Access program during the second half of their junior year of high school. BL collects a 
substantial amount of self-reported academic and demographic information from students through 
the application, and verifies self-reported family income and academic performance information 
through tax records and high school transcripts, respectively. Students are eligible for BL if their 
families make less than 200 percent of the federal poverty guidelines and if they have a high school 
GPA of 2.5 or higher.!4 The program serves a sizeable share of the students that meet the program 
eligibility requirements in in its focal cities — 60-70 percent in the Boston area (i.e., the Boston 
and Worcester sites).!° The high rate of take up suggests that were BL to scale, it would likely 
reach most eligible students in small/medium cities. While BL was not operating in New York at 
the time of the market analysis, the firm’s estimates suggest that BL’s reach in New York only 
serves a small share of the roughly 12,000 students in the city who meet the program eligibility 
requirements. 

BL advisors begin working with admitted students between the end of their junior year and 
the start of their senior year of high school. Advisors work full time. All advisors have a college 
degree and 17 percent have a master’s degree. Most advisors are female (75 percent) with roughly 
a quarter black and a quarter Hispanic. The median advisor age is 26. Advisors have an average 
caseload of 50-60 students and meet with each student for an hour every three or four weeks during 
senior year, at BL’s office in each community. BL advisors provide comprehensive college and 
financial aid support for students, ranging from creating lists of potential schools and supporting 
students with essay writing and application completion, to applying for financial aid, searching 
for scholarships, interpreting financial aid award letters, and selecting a college or university that 
aligns with a student’s goals and circumstances. Outside of direct time with students, advisors 


work on additional advising-related activities, like developing customized college lists for students, 


™4 An earlier regression discontinuity design evaluation of the BL program in the Boston area found suggestive 
evidence that BL positively impacts college enrollment (Castleman and Goodman, 2016). The paper does not 
examine impacts on degree attainment, and the results are imprecisely estimated (including enrollment effects as 
small as negative 20 pp and as large as 40 percentage points), rely on a manipulable running variable, and are local 
to the 2.5 GPA threshold determining eligibility for the program. 

15This estimate is based on a market analysis BL contracted a consulting firm to conduct in its Massachusetts 
markets. 


reviewing students’ college essays, and analyzing students’ award letters. We provide descriptive 
statistics on the frequency and trends in advisor engagement with students, and the topics advisors 
and students discuss, in the “Exploring Access Advisor Effectiveness” section below. 

Once students have chosen where to enroll in college, students who plan to attend one of BL’s 
target institutions are invited to continue into the Success program.!® The BL target institutions 
are primarily moderately selective four-year colleges and universities, approximately two-thirds of 
which are public institutions. Campus-based advisors at each target institution meet regularly 
with students once they have matriculated in college; first-year students meet with advisors ap- 
proximately three to four times per semester, while older students meet with an advisor twice a 
semester on average.!’Success advisors provide a combination of academic support (e.g. course 
selection and making use of advising and tutoring services) and social support (e.g. helping stu- 
dents adjust to a new environment, getting involved with activities and student groups), and advise 


students on how to balance academic, work, social, and family commitments. 


3 Experimental Design 


We collaborated with BL staff to modify its student application processes in the spring of 2014 and 
spring of 2015 to incorporate a lottery design into BL’s selection of applicants. In the spring of 2014 
BL accepted applications in two waves: one application window closed at the end of May and the 
other application window closed at the end of August 2014, and in 2015 BL accepted applications 
in one wave, at the end of August 2015. Among students who met the BL eligibility criteria (GPA 
of at least 2.5 and family income below 200 percent of the poverty line), we randomized students 
to either receive an offer to participate in the BL Access advising program or to be in a control 
group that did not receive any BL services. In each site BL had minimum commitments to its 
funders and community partners of the number of students it had to serve, which are reflected in 


the treatment /control ratios we report in Table A2.'% 


16 Appendix Table Al shows the list of encouraged institutions at each BL site 

17 Advisors adjust meeting frequency based on student need, meeting more regularly with students who are experi- 
encing some form of challenge in college. Success advisors typically serve students across 2-3 different campuses, and 
work with 30-40 students per campus. 

'8The first experimental cohort included 1,429 students, while the second experimental cohort included 993 students. 


3.1 Data 


Our data come from four primary sources: the BL application, BL advisor interaction data, two 
surveys we conducted with students during the spring of their senior year in high school and the 
fall after high school graduation, and the National Student Clearinghouse, from which we obtained 
college enrollment and degree attainment data. The BL application collects rich student-level 
baseline information, including race/ethnicity, gender, whether the student is the first in their 
family to go to college, whether they were working with another college access organization at 
the time they applied for BL, their high school GPA and SAT/ACT scores (if they had taken the 
exam), family income, and whether they had a sibling who had participated in BL. The interaction 
data contains detailed information on each interaction students had with an advisor, including the 
topic discussed, assistance the advisor provided with this topic, and narrative comments from the 
advisor about their interaction with students. Our spring of senior year survey asked students 
where they applied to college and whether they had been accepted to each institution; whether 
and when students applied for financial aid; whether students received assistance reviewing their 
financial aid award letters; and a series of questions about factors influencing students’ decisions 
about whether and where to enroll in college. Our fall after high school survey asked about students’ 
enrollment intensity, campus engagement, course taking, and employment. The National Student 
Clearinghouse (NSC) provides student*term-level college enrollment and degree attainment data, 


with coverage across more than 97 percent of college enrollments in the country. 


3.2 Baseline Equivalence 


In Table 1, we report results from models in which we regress student-level baseline characteristics 
on the treatment indicator and site by cohort fixed effects. Across 20 baseline measures we only 
find 2 statistically significant difference between the treatment and control group at the 10 percent 


level, which is probabilistically what we would expect given the number of tests we conduct. 


4 Empirical Strategy and Results 


We estimate the effects of an offer to participate in BL on college enrollment, enrollment quality, 
persistence, and degree attainment. As the proportion assigned to treatment varied by site and 
cohort, we control for site by cohort fixed effects. In most specifications we condition on covariates 


to increase precision. Our basic specification is: 


Yy, = at BX; + OTreatment; + S- Vilig PE; (1) 
J 


where y; is generally an enrollment outcome for individual 7 and X; includes baseline demographic 
controls (gender, race, citizenship), measures of family resources and background (parents’ AGI, 
parental employment status, household size, first generation status, whether sibling went to col- 
lege), measures of aptitude (standardized GPA, state standardized test scores), and measures of 
college guidance resources (whether student is working with another advising organization, whether 
sibling participated in BL). The l,; are site by cohort fixed effects. These are included because the 
probability of being assigned to treatment varies by site and cohort. The coefficient of interest is 


6, which is the intention to treat (ITT) estimate. 


4.1 Enrollment 


Table 2 contains our estimates of the impact of BL on students’ enrollment and enrollment quality. 
The point estimate in the first row of column (2) shows that assignment to treatment increases 
the likelihood of college enrollment by 5.3 percentage points (6.5 percent). While these overall 
enrollment effects are similar to most rigorous estimates of the effects of pure advising programs, 
the primary focus of the BL model is promoting four-year and higher-quality college enrollment, 
and in turn, bachelor’s degree attainment. Estimates in the second row of Table 2 indicate even 
larger effects on four-year enrollment, with a 9.1 percentage point increase; this is a 13 percent 
increase relative to four-year enrollment in the control group. As expected given BL’s emphasis 
on four-year enrollment, estimates in the third row indicate a reduction in two-year enrollment (4 
percentage points) contributed to the rise in four-year enrollment. 


The remaining rows in Table 2 estimate the impact of BL on the quality of institutions at which 


10 


students enroll. Treated students are substantially more likely to attend institutions with higher 
graduation rates, lower default rates, higher post-enrollment earnings, and higher mobility among 
attendees.!9 For instance, BL students are 11 percentage points more likely to attend institutions 
with above-median earnings for graduates; this represents a 17 percent increase relative to the 
control.?° 

While college enrollment has increased dramatically over the last thirty years, the rate of four 
year college completion has barely increased. Many college students fail to make it past the first 
year, underscoring the importance of evaluating the effects of intensive advising past the point of 
initial enrollment. 

In Figure 1 we compare enrollment rates over time for the BL students compared to the control 
group. We measure enrollment each term (Fall or Spring) starting one year after high school and 
continuing through five years after high school. The top panel presents overall enrollment rates 
over time while the bottom panel presents enrollment rates over time at four-year institutions. 
Starting in the first Fall semester after high school, and continuing through the Spring semester 
four years after high school, BL students maintain a substantial and consistently (%5 percentage 
point) higher rate of enrollment than control students (panel (a)). We observe a very similar but 
more pronounced pattern of increased four-year enrollment over time, with BL students maintaining 
a large and consistently higher rate of enrollment (+10 percentage points) starting the first Fall 
after high school and continuing four years after high school (panel (b)). In both cases, we observe 
virtually no difference in enrollment starting five years after high school as a sizeable share of 


students assigned to BL earn bachelor’s degrees and leave college.?! 


Specifically, we estimate effects on whether an individual is initially observed enrolled at a four-year institution 
with an above median 6-year graduation rate (53.8), with a student loan default rate below the median (0.07), with 
earnings above the median for all college enrollees ($35,800) in Chetty et al. (2017), and with “mobility” above the 
median college in Chetty et al. (2017). 

20The estimates in Table 2 are essentially unchanged upon incorporation of covariates or the estimation of average 
marginal effects from specifications that interact risk set indicators with treatment status. 

21 As we show in Appendix Figures Al and A2, these patterns of overall and four-year enrollment over time are 
quite similar for both experimental cohorts. 
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4.2 Degree Attainment 


In Table 3 we present estimates of BL’s impact on degree attainment. The point estimates in 
column 2 show that, starting four years after high school, BL generates large increases in the share 
of students that earn a bachelor’s degree. BL students are 6.2 percentage points more likely to 
earn a bachelor’s degree within four years after high school; this represents a 23 percent increase 
relative to the control group mean of 26.8 percent. By five years after high school, this grows 
to a 7.6 percentage point effect relative to the control mean of 47.1 percent. By six years after 
high school (which we observe just for the first experimental cohort) the control mean bachelor’s 
degree attainment rate rises to 52.8 percent. Nonetheless, BL students remain much more likely 
(9.6 percentage points) to earn a bachelor’s degree. 

The next four rows in Table 3 show that BL’s impacts on degree attainment result mainly from 
increasing the share of students who graduate from high-quality institutions, those with higher 
graduation rates, lower default rates, higher average earnings among their graduates, and higher 
mobility rates. For instance, BL students were 7.6 percentage points more likely to graduate from 
an institution with above-median average earnings for graduates, a 15 percent increase relative to 
the control group. 

The final two rows in Table 3 show that a moderate share of BL’s impact on bachelor’s degree 
attainment likely results from diverting students who would otherwise have attended two-year 
institutions and received associate’s degrees to instead enroll at four-year institutions and complete 
their bachelor’s degree. For instance, BL students were 3.0 percentage points less likely to earn an 
associate’s degree within five years. 

For context, these bachelor’s attainment effects are similar in size to those estimated in a 
recent experimental evaluation of Buffett Scholars, a program that provides merit-based aid for 
low-income students. While the attainment effects are similar in size, the Buffett program cost per 
treated applicant is roughly eight times the amount spent per BL-offered student. The marginal 
degrees generated by BL advising appear to be larger per direct dollar spent than that of any large 
financial aid program that has been rigorously evaluated (Figure 1). 


Indeed, the effects of BL on bachelor’s degree attainment appear to be as large or larger than 


12 


any intervention for which researchers have rigorously estimated treatment effects. This includes 
the estimated effects of early childhood education on bachelor’s degree attainment, which range 
from null effects for Perry Preschool to null to somewhat similar effect sizes as what we estimate 
for Head Start (Anderson 2008; Thompson 2018; Bailey, Sun, Timpe 2020). But these programs 
are considerably more expensive, particularly when one considers that the investment is made 14 or 
more years in advance of college enrollment. Assignment to a smaller class in primary school also 
produces much smaller effects on bachelor’s degree attainment (0.9 pp), and does so at significantly 
greater cost (roughly $14,000 in 2015 dollars) (Dynarski, Hyman, Schanzenbach 2013).72 

In Figure 2, we explore the extent of heterogeneity in effects across common subgroups defined 
by race, gender, high school performance, access to alternative college access programs, and family 
resources. While we see some suggestive evidence of smaller treatment effects for those with below- 
median high school GPAs in panel (a), this difference appears to be driven by low levels of degree 
attainment in this group four years after high-school graduation. Looking five years after high- 
school graduation, BL’s positive impacts on bachelor’s degree attainment are quite consistent across 


student groups. 


5 Mechanisms 


5.1 Explaining Bachelor’s Treatment Effects Using Causal Forests 


We have shown in the prior sections that participation in BL college advising generates large 
increases in the share of low-income students that attend higher-quality four-year institutions and 
that earn bachelor’s degrees within 5-6 years of high school. It could be the case that shifting 
students to attend higher-quality institutions results in the higher rates of degree attainment. But 
it is also possible that the students induced to attend higher-quality colleges and universities were 
inframarginal to bachelor’s degree attainment and that BL is affecting degree completion among 


a different population of students (e.g., through the ongoing Success advising provided to many 


?2Tcreases in school quality also appear to produced smaller increases in bachelor’s degree attainment (Deming et 
al. 2011). Attending a first-choice school in Charlotte-Mecklenberg produced smaller effects on degree attainment (4.7 
pp), although it is difficult to scale these effects by an associated cost. Lottery-based evaluations of charter middle 
schools suggest no increases in bachelor’s degree attainment (https://files.eric.ed.gov/fulltext /ED594043.pdf). Even 
charter schools that produced very large increases in academic achievement and on-time benchmarks failed to produce 
persistent increases in college attainment (Dobbie and Fryer 2015). 
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students). 

One traditional approach to distinguishing these explanations is to examine patterns of sub- 
group heterogeneity to see if the subgroups with larger treatment effects on initial enrollment quality 
are also the subgroups with larger treatment effects on bachelor’s degree attainment. This type 
of analysis is usually conducted across commonly defined subgroups, e.g. race, gender, baseline 
level of achievement, etc. In our case, the lack of heterogeneity across these subgroups results in 
this approach being uninformative about the role of institution quality in mediating the bachelor’s 
degree effects that we estimate. Instead, we use recently developed causal forest methods in a novel 
way to investigate the extent to which we can attribute increases in bachelor’s degree attainment 
to BL’s impact on students attending higher-quality institutions. 

Causal forests are a machine learning-driven enhancement to traditional methods for investi- 
gating treatment heterogeneity (Athey and Imbens, 2016; Athey and Wager, 2019). Rather than 
rely on researchers to specify subgroups to include in heterogeneity analysis, causal forest meth- 
ods use machine learning to investigate the complex relationships between baseline covariates and 
treatment effects. The intuition behind the method is to create “trees” that iteratively split the 
sample in ways that maximize treatment effect heterogeneity. For example, one tree might split 
on gender and then GPA below or above 3.0, and so on. Individuals in the resulting “leaves” of 
the tree share similar combinations of baseline covariates that are predictive of the treatment ef- 
fect associated with their group. To prevent overfitting, the treatment effect associated with these 
“leaves” is estimated using a holdout sample that was not used in the production of the tree. Using 
a new random training (and hold out) sample each time, the algorithm produces a series of these 
trees that are referred to as a “forest”. This “forest” can then be used to generate personalized 
treatment effects (PTEs) for each individual by averaging across the treatment estimate associated 
with each “leaf” that an individual falls into in each tree. These PTEs indicate the magnitude of 
impact the treatment has for an individual subject (on a particular outcome) based on their observ- 
able characteristics. While researchers have used causal forests to enhance treatment heterogeneity 
analysis (e.g., Davis and Heller, 2017), applications of causal forests to educational interventions 
remain very limited, and we are unaware of any that use the method to explore mechanisms in the 


fashion we pursue in this paper. 
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To investigate whether BL’s impacts on enrollment quality are resulting in higher rates of 
degree attainment, we estimate separate PTEs for two outcomes: (1) enrollment at a four-year 
institution with above-median graduation rates and (2) bachelor’s degree attainment within five 
years. In Figure 5, we display the relationship between the PTEs we estimate for attendance at 
higher-quality institutions (X axis) and the PTEs we estimate for bachelor’s degree attainment 
(Y axis).?? We observe a strong positive relationship between the PTEs: students for whom BL 
has the largest positive personalized treatment effects on enrollment quality tend to be the same 
students for whom BL has the largest PTEs on bachelor’s degree attainment.24_ In Figure A4 
we further leverage our causal forest analysis by estimating (using equation (1)) BL’s impact on 
bachelor’s degree attainment by whether students had below- or above-median PTEs on enrollment 
at four-year institutions with above-median graduation rates. We estimate large (+15 percentage 
point) impacts on bachelor’s degree attainment for students with above-median PTEs on enrollment 
quality, and no effect on attainment for students with below-median PTEs on enrollment quality. 
In other words, assignment to BL advising only appears to generate additional degrees for students 
who are likely to enroll in a high quality college as a result of BL advising. 

This analysis suggests shifting students to enroll at higher-quality institutions is the primary 
driver of BL’s impact on bachelor’s degree attainment. To supplement this analysis, we present 
in Appendix Table A3 a mediation analysis in which we include indicators for students’ initial 
enrollment and enrollment quality in our basic specification to estimate treatment effects. The 
coefficient on treatment is substantially attenuated, to 1.6 percentage points (n.s.), which suggests 
that most of BL’s impact on attainment stems from the effect Access program advisors have on 
students’ college choices.2° While suggestive, the modest positive coefficient may indicate that the 
ongoing advising provided by Success advisors does help some students persist to degree. Consistent 
with this, we do see some evidence that individuals who, based on their covariates, are likely to 


attend BL target institutions absent treatment have larger PTEs (and actual estimated treatment 


3Tn the figure, we collapse the PTEs for bachelor’s degree attainment into means within bins of size 0.02. The size 
of each dot reflects the sample size represented. 

24 Appendix Figure A5 shows a similar but weaker relationship between the individual PTEs we estimate for 
attendance at a four-year institution (X axis) and the individual PTEs we estimate for bachelor’s degree attainment 
(Y axis). 

25 Interpreting this result seriously requires strong assumptions regarding the presence of confounders along multiple 
margins. With that caveat in mind, we still think the estimates are informative. 
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effects) in terms of 5-year BA attainment; this is true despite these individuals having smaller 
PTEs (and actual estimated treatment effects) on the four-year enrollment margin. In other words, 
among those likely to attend BL target schools, assignment to treatment is generating more marginal 
degrees per marginal enrollee, suggesting that the Success program may contribute somewhat. to 
increased persistence, in addition to the important role of Access counselors in shaping school 


choice. 


5.2 Exploring Access Advisor Effectiveness 


How are advisors so effective in shaping school choice? In the final results sections we attempt to 
more rigorously measure what advisors do and what appears to be effective, leveraging survey data, 
rich student-advisor interaction data, and the quasi-random assignment of students to advisors. 

BL maintains detailed data on advisor-student interactions. Advisors record a note detailing 
the date, mode of contact, and purpose of each interaction. They also enter a written summary 
of the substance of the meeting. Appendix Table A4 contains summary statistics generated from 
these data for the period between the beginning of the Access advising program (May of student’s 
Junior year of high school) and the transition period to college (August after a student’s Senior year 
of high school). As seen in the table, nearly every student assigned to treatment (97 percent) had 
at least one interaction with an advisor during this period. While nearly every student (95 percent) 
had an in-office meeting with an advisor, only a third talked to an advisor on the phone. Over the 
15-month period, advisors interacted with students an average of 13 times, with the majority of 
these interactions occurring as in-person meetings in the advisor’s office. 

Figure 2 illustrates the frequency of student-advisor interactions by months since the beginning 
of the advising program. As the BL model begins at the end of a student’s junior year of high 
school, month 0 is set to equal May of 2014 for the high school class of 2015 and May of 2015 for 
the high school class of 2016. As illustrated in the figure, advisors begin interacting with students 
in the summer after their junior year and continue interacting with most students at high levels 
into the spring of the following year. We estimate that advisors spend an average of 10 to 15 hours 
working directly with each student 


In addition to illustrating the high and persistent level of student-advisor interaction, the 
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interaction data provide a way to quantify what advisors are spending time on during these meet- 
ings. The bottom third of Appendix Table A4 indicates that most meetings involve working on 
applications (3.47 meetings per student) or financial aid (2.03 meetings per student). 

In general, advisors try to guide students to choose schools with relatively low costs and high 
graduation rates. The set of schools that possess these traits tend to coincide with the set of target 
colleges where BL has a continued advising presence. 

Whereas the administrative data provide a good indication of how advisors spend their time 
helping students, they provide relatively little indication of the specific changes in students’ actions, 
behaviors, and/or attitudes that led to the pronounced impacts we observe on college enrollment 
and persistence. To better understand the channels through which the BL advising may have 
affected students’ college decisions and outcomes, we turn to survey data. 

We conducted a survey of both treatment and control group students in the first cohort during 
their spring of the senior year of high school (2015). We asked about students’ college and financial 
aid application decisions and behaviors; where they had been accepted as of the time of the survey; 
and the sources of advising and support students relied on when making college and financial aid 
decisions (for treatment group students, this included questions about their BL advisor). Approx- 
imately 56 percent of students responded to the survey, with roughly equal response rates among 
treatment and control group students.7° 

One interesting finding that emerges is that nearly all survey respondents in the control group 
—those who applied for BL but were not selected to participate — applied to college and for 
financial aid, even in the absence of BL advising (Table 4). This suggests that control group 
students were able to access college planning guidance and support from other sources, or had 
sufficient motivation and college aspirations to complete these tasks independently. Students in 
both experimental groups also applied to a large volume of colleges and universities — 10 on 
average for control group students and 13 on average for students in the treatment group. 


In terms of students’ responses about sources of college and financial aid advising, treatment 


26The response rate for the control group was 0.558, with a 0.016 (se 0.029) coefficient on a treatment indicator 
variable, controlling for site by cohort indicators. Appendix Table A5 suggests little selection or differential selection 
into survey response. As seen in the table, observables of respondents are similar to those of the full sample. Similarly, 
the characteristics of treated respondents are broadly similar to control respondents. 
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students rate BL advising as the most important source of guidance; 58 percent of treatment 
students indicated that BL advising was “very important” in their application and decision process. 
In contrast, only 21 percent of control group students indicated that “staff at other college access 
programs” were very important. Both groups ranked support from parents (* 60 percent), advisors 
(= 50 percent), and teachers (= 30 percent) as very important. 

Interestingly, among students who ranked parents, advisors, or teachers as important, treat- 
ment students were less likely to say they discussed college-related issues (e.g., which colleges to 
apply to or how to apply for financial aid) with these other adults. This suggests treatment stu- 
dents received more guidance on these topics from BL advisors, and perhaps felt less need to turn 


to other (and potentially less-informed) sources of advising for this information. 


5.3 Do Effects Vary across Advisors? 


One question related to mechanisms and to the potential scalability of BL is whether the large 
observed treatment effects on enrollment, persistence, and degree attainment vary across advisors. 
Understanding the extent to which effects vary across advisor characteristics/behavior provides 
insight into the channels through which BL is influencing behavior as well as the extent to which 
the BL model can be successfully replicated in other settings.?" 

Figure 6 plots estimates of advisor fixed effects on college enrollment and four-year college 
enrollment. As seen in the Figure 6, 29 out of 30 advisors have a positive fixed effect on BL’s 
focal outcome, four-year enrollment. This is further evidence of the potential scalability of the BL 
model.?® 

Of course, the preponderance of positive advisor fixed effects may reflect some sorting of 
students who need the most help to the most effective advisors. The inclusion of baseline covariates, 
suggests that this is not the case. After controlling for a rich set of student baseline covariates, 29 


out of 30 have a positive effect on four-year enrollment. 


Despite the suggestion of positive impacts across nearly all advisors, these figures do not 


27Tf only advisors with certain characteristics/behaviors are effective and advisors with these characteristics are in 
short supply, it would be more difficult to scale the program. 

?8We note that it is, of course, possible that the quality of advisors falls with expansion, but argue that the 
consistency of effects across current advisors suggests the importance of the BL model to ensuring consistent delivery 
of services. Indeed, despite quickly scaling into the NYC area we observe similar effects there. 
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necessarily indicate the causal effect of particular advisor characteristics on student outcomes. 
Advisors in high schools and other college success organizations often have some say in who they 
counsel. Advisor preferences (and thus characteristics) could therefore be correlated with the ability, 
family background, and motivation of their students. Similarly, many college success organizations 
intentionally match advisors to students based on similarity of backgrounds or interests, hoping 
that shared experiences will result in a better match and a higher likelihood of helping the student. 

BL staff follow a different approach, essentially assigning students to advisors at random. BL 
staff describe the Access advisor assignment process as a “pretty blind assignment to fill each 
advisor’s caseload (when they come in and who is available to meet with them).” While no formal 
randomization procedure is followed, this discussion suggests that student assignment to advisor 
may be as good as random. 

We explore this notion more formally in Appendix B, providing evidence consistent with the 
quasi-random assignment of students to advisors. Having established quasi-random assignment, 
we proceed with an investigation of the effects of advisor characteristics and behaviors on student 
success (details in Appendix B). We find no observable relationship between advisor gender or race 
and student success (Table 5).2? While interesting in its own right, the lack of heterogeneity across 
different types of advisors further suggests the potential scalability of the program.?? The BL 
model appears effective across nearly all advisors, and there is little relationship between advisor 


characteristics and advisor effectiveness. 


6 Discussion 


Through a multi-cohort, multi-site RCT, we provide robust evidence that intensive advising gener- 


ates large increases in degree attainment, with large increses in degrees received from higher quality 


2°There is some suggestive evidence that assigning a student to an advisor who tends to have more application 
meetings with their assignees may increase the likelihood that that student goes to college, suggesting that BL’s 
increased focus on this aspect of advising may be important. While it is clear that advisors that have more application 
meetings are more effective, the results merely suggest that the extent of interaction is causing the higher enrollment 
rates. It may be that advisors who have more application meetings have some other characteristics that makes them 
a better advisor. 

3°We have also investigated the presence of racial or gender interaction effects (for example, does a black student 
benefit more from a black advisor). While power limits our ability to draw strong conclusions, we find no evidence 
of important interactions of this type. 
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institutions based on traditional metrics (e.g., graduation rate, cohort loan default rate, and post- 
enrollment earnings) and a newer measure of social mobility, the fraction of low-income students 
moved from the bottom to the top income quintile. Given declining social mobility over time, inten- 
sive advising appears to be an effective policy strategy to promote greater economic opportunity 
for economically-disadvantaged, academically college-ready students. While the observed degree 
effects are quite consistent across different types of students, the fact that BL primarily serves 
students of color furthermore suggests that substantial expansion of the BL model could contribute 
to increased racial equity and mobility in the U.S. 

We use novel casual forest methods to demonstrate that BL’s impact on attainment is gener- 
ated primarily by shifting students to enroll at higher-quality institutions. This is consistent with 
analyses of administrative and survey data that underscore the role of advisors in shaping student 
application and college choices. While we cannot rule out the role of the ongoing advising provided 
by BL’s Success advisors in promoting persistence and attainment, the evidence suggests that they 
perhaps play a modest role in generating additional degrees. 

The consistency of our results across sites, cohorts, advisors, and students indicate that the BL 
model may provide a scalable solution to reducing the income gap in college enrollment and success. 
We find that BL generates large impacts across multiple program sites operating in different states 
under local program leadership. The New York site had been in operation for only a few years prior 
to the RCT. Large positive effects of the BL model there provide direct evidence of scalability and 
suggest that the program reaches maturity and efficacy more rapidly than many other programs. 

We believe it is particularly noteworthy how consistent BL advisors are at improving student 
outcomes. From a scalability perspective this is highly important, since it suggests that a combina- 
tion of coherent organizational leadership, successful staff recruitment and training, and effective 
curriculum are driving the results we observe, rather than a handful of particularly strong advisors 
who may be hard to identify and recruit in other contexts. Of course, it is possible that the quality 
of BL advisors could fall with program expansion in an area, but high rates of take up in the Boston 
area (BL reaches 60-70% of eligible students) suggest that advisor quality can be maintained. 

Further supporting the notion that BL may be scalable, the colleges targeted by BL account 


for a substantial (40%) and responsive share of nearby four-year enrollment suggesting that BL is 
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not just crowding out other students.*! An open question though is whether the effects are likely to 
be similar if BL access was expanded to additional individuals who are less likely to apply. While 
we cannot answer this question definitively, we think it is likely for two reasons. First, eligible non- 
applicants tend to be somewhat more disadvantaged, a group for which we find larger effects among 
applicants. Second, our estimated treatment effects are generated off of cohorts of applicants for 
which BL aggressively recruited students to meet sample size targets for the experiment, sample 
sizes that were substantially larger than the number of students served in prior cohorts. In some 
sense then, the estimated average treatment effects incorporate the treatment effects for individuals 
who would not have applied absent the experiment. 

Taken collectively, these results suggest that the BL model has the potential to maintain its 
positive impact with diverse populations in numerous settings. It is also impressive that BL has 
generated large and growing impacts on students’ postsecondary outcomes given that two of the 
markets in which it operates (New York and Boston) are fairly saturated with other college advising 
organizations. BL’s impacts could be even larger if applied in communities where students have 


32 Even assuming similar effects in other 


little or no existing access to college advising supports. 
communities, broad adoption of the BL model would substantially reduce attainment gaps; the 
estimated impact of advising on bachelor’s degree attainment is roughly as large as the (conditional 
on aptitude) gap in degree attainment between children from families in the first and fourth quartile 
of the income distribution. 

As we describe earlier and show in Figure 1, the BL model is also more cost-effective than 


rigorously evaluated aid programs that provide students with additional financial assistance for 


college. While maintaining implementation quality at scale for BL may be more challenging than 


3'Between 2003 and 2014, enrollment at BL target colleges rose nearly 30 percent (7,690 students), while enrollment 
at Ivy league institutions in the same areas rose roughly 3.6 percent (142 students). Were BL primarily targeting 
students towards highly selective institutions with limited changes to total enrollment over time (e.g., the Ivies), 
the concern about crowd out would be more pronounced. BL target institutions also appear to increase capacity in 
response to demand; regressing log enrollment on log applications in first differences indicates that enrollment rises 
by 5.4 percent for every 10 percent increase in applications. In contrast, enrollment levels at elite and Ivy league 
institutions are unresponsive to the number of applications. The point estimate for BL target institutions is 0.54 (se, 
0.043), while that for Ivy and elite institutions is -0.02 (se, 0.021). Based on authors’ calculations using IPEDS data 
restricted to commuting zones containing the BL target colleges. 

°° Appendix Table A6 illustrates similar treatment effects on enrollment and bachelor’s degree attainment by site. 
The Boston area includes the Boston and Worcester sites. The latter is much smaller (337 students) so for precision 
we combine with the Boston site. However, impacts on BA within five years are very similar for the Boston (7.2 pp) 
and the Worcester sites (7.4 pp). 
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for aid programs, the consistency of BL’s impacts across sites, student groups, and advisors suggests 
that providing cities with a combination of financial and technical assistance to replicate the model 
could be a more efficient approach to increasing bachelor’s degree attainment than offering students 
additional aid. Indeed, the advising model appears to be as cost-effective at increasing bachelor’s 


degree attainment as any rigorougly evaluated policy of which we are aware. 
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Figure 1: Bachelor’s Degrees Generated per $1,000 of Direct Cost: Advising and Financial Aid 
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Note: Estimates produced by dividing reported estimate of the effect of a program on bachelors degree 
attainment by the corresponding cost (inflated to 2016 dollars). 


Figure 2: Advisor Interaction Patterns over Time 
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Note: Statistics derived from BL data. Month 0 is May of each high school class’ junior year. 
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Figure 3: College Enrollment Over Time 
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Note: Statistics derived from BL and NSC data. Year 1 is the academic year beginning in the fall after 
each high school class’ senior year. Treatment line points adds estimated treatment effect from equation (1) 
to the control mean. 
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Figure 4: Bachelor’s Degree Attainment: Heterogeneity 
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Note: Statistics derived from BL and NSC data. Treatment estimates provided by estimating equation (1) 
within each subgroup. 


Figure 5: The Relationship Between Treatment Effects on High Quality Enrollment and 5-Year 
Bachelor’s Degree Receipt 
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Note: Statistics derived from BL data. PTEs produced using covariates included in main specification. See 
text for additional details. We collapse the individual PTEs for bachelor’s degree attainment into means 
within bins of size 0.02. The size of each dot reflects the sample size represented. 


29 


Figure 6: Advisor Fixed Effect Estimates 
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Note: Estimates derived from basic specification but replacing treatment indicator with advisor fixed effects. 


30 


Table 1: Descriptive Statistics and Randomization Tests 


Control Mean ‘Treatment 


(1) (2) 
Female 0.697 0.004 
(0.021) 
Black 0.302 0.022 
(0.021) 
Hispanic 0.325 -0.008 
(0.021) 
Asian 0.246 -0.009 
(0.020) 
Other Race 0.094 0.001 
(0.014 
Citizen 0.787 -0.039** 
(0.019) 
Verified GPA 3.264 -0.004 
(0.027) 
Parent AGI 22520 393 
(840) 
Household Size 4.26 -0.003 
(0.074) 
Mom Employed 0.641 .005 
(0.023) 
Mom Employed (missing) 0.144 -0.007 
(0.016) 
Dad Employed 0.435 0.053** 
024 
Dad Employed (missing) 0.446 -0.004 
023 
First Generation 0.811 .000 
.019 
Sibling College 0.389 -0.004 
023 
Sibling College (missing) 0.059 -0.011 
.010 
Sibling Bottom Line 0.075 001 
013 
Sibling Bottom Line (missing) 0.074 -0.001 
(0.012) 
Other Program 0.444 -0.009 
(.022) 
Observations 2422 


Note: Column (1) contains control group means. Each cell 
in column (2) contains a coefficient from a separate regres- 
sion of the observed characteristics on a treatment indicator 
variable, controlling for site by cohort (i.e., risk set) indica- 
tors. Robust standard errors in parentheses. * (p<0.10) 
**(p<0.05), ***(p<0.01). 
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Table 2: Effects on Enrollment in College 


Control Mean ‘Treatment 
(1) (2) 
Any College 0.822 0.053*** 
(0.017) 
4-Year College 0.697 0.0914*** 
(0.020) 
2-Year College 0.128 -0.040*** 
(0.014) 
High Grad Rate 0.275 0.088*** 
(0.020) 
Low Default 0.291 0.072*** 
(0.020) 
High Earnings 0.631 0.110*** 
(0.020) 
High Mobility 0.608 0.103*** 
(0.020) 
BL Target College 0.327 0.101*** 
(0.022) 
Observations 2422 


Note: Column (1) contains control group means. 
Each cell in column (2) contains a coefficient 
from a separate regression of each outcome vari- 
able (indicated by the row title) on a treatment 
indicator variable, controlling for site by cohort 
(ie., risk set) indicators as well as the covari- 
ates indicated in Table 1. Robust standard er- 
* (p<0.10) **(p<0.05), 


rors in parentheses. 


***(p<0.01). 
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Table 3: Effects on Degree Attainment 


Control Mean ‘Treatment 


() (2) 
BA Degree (4 Years) 0.268 0.062*** 
(.020) 
BA Degree (5 Years) 0.471 0.076*** 
(0.022) 
BA Degree (6 Years)* 0.528 0.096*** 
(0.027) 
High Grad Rate BA 0.252 0.082*** 
(0.0189) 
Low Default BA 0.250 0.072*** 
(0.023) 
High Earnings BA 0.502 0.076*** 
(0.026) 
High Mobility BA 0.480 0.083*** 
(0.026) 
AA Degree (4 Years) 0.106 -0.033** 
(.013) 
AA Degree (5 Years) 0.128 -0.030** 
(0.014) 
Observations 2422 


Note: Column (1) contains control group means. 
Each cell in column (2) contains a coefficient from 
a separate regression of each outcome variable (in- 
dicated by the row title) on a treatment indicator 
variable, controlling for site by cohort (i.e., risk set) 
indicators as well as the covariates indicated in Ta- 
ble 1. a: BA Degree (6 Years)*) is only estimated for 
Cohort 1. Robust standard errors in parentheses. * 
(p<0.10) **(p<0.05), ***(p<0.01). 
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Table 4: Student Completion of College and Financial Aid Milestones (Cohort 1) 


Control Mean ‘Treatment 


(1) (2) 
Proportion Applying 0.988 0.009 
(0.007) 
Number of Applications 9.75 2.91*** 
(0.336) 
Costs Important 0.50 0.09* 
(0.05) 
Filled Out FAFSA 0.97 0.017 
(0.05) 
Met to Review Award Letter 0.66 0.18* 
(0.09) 
College Access Advisor Important 0.21 0.37* 
(0.22) 
Observations 813 


Note: Statistics derived from survey of individuals in cohort 1. 
See text for additional details. Column (1) contains control group 
means. Each cell in column (2) contains a coefficient from a sep- 
arate regression of each variable (indicated by the row title) on a 
treatment indicator variable, controlling for site by cohort (i.e., 
risk set) indicators as well as the covariates indicated in Table 1. 
Robust standard errors in parentheses. (p<0.10) **(p<0.05), 
***(p<0.01). 
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Table 5: Relationship Between Advisor Characteristics and Enrollment Outcomes 


Enrolled Enrolled 4-Year Enrolled High Grad BA 5 Years 
(1) (2) (3) (4) (5) (6) (7) (8) 
Counselor Characteristics 

Female 0.011 0.011 -0.006 -0.012 0.004 0.000 0.010 -0.001 
(0.027) (0.027) (0.035) (0.034 0.042 (0.041) (0.043) (0.042) 
Black -0.041 -0.042 -0.045 -0.048 = -0.023 -0.013 -0.010  -0.007 
(0.029) (0.029) (0.037) (0.037 0.045 (0.044) (0.046) (0.045) 
White -0.033 -0.032 -0.034 -0.032  -0.018 -0.010 -0.012 = -0.003 
(0.029) (0.029) (0.037) (0.037 0.045 (0.044) (0.046) (0.045) 
Hispanic -0.020 -0.019 -0.024 -0.023 = -0.008 -0.010 0.010 0.018 
(0.028) (0.028) (0.036) (0.035 0.043 (0.042) (0.044) (0.043) 
Application Meetings 0.064** 0.066** 0.055 0.054 -0.009 -0.027 0.005 -0.008 
(0.032) (0.032) (0.040) (0.040 0.049 (0.048) (0.050) (0.049) 
Financial Aid Meetings -0.033 -0.037 -0.043 -0.050 -0.037 -0.034 -0.016 -0.028 
(0.044) (0.044) (0.055) (0.055 0.067 (0.066) (0.069) (0.068) 

Covariates xX x xX x 


Note: Each column contains estimates from a separate regression of a dependent variable (in columns) on a 
set of advisor characteristics. Application meetings and financial aid meetings variables provide a measure of 
the average number of meetings of each type per student for each advisor. The variable is constructed using a 
leave one out procedure, so that each individual is assigned the average number of meetings occurring between 


every other student with the same advisor. Robust standard errors in parentheses. 


**(D<0.01). 


* (p<0.10) **(p<0.05), 


Appendix A: Supplemental Tables 
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Figure Al: College Enrollment Over Time: First Cohort 
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(b) Enrolled Four-Year 


Note: Statistics derived from BL data. Year 1 is the academic year beginning in the fall after each high 
school class’ senior year. Treatment line points adds estimated treatment effect from equation (1) to the 
control mean. Sample restricted to cohort 1. 
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Figure A2: College Enrollment Over Time: Second Cohort 


Fall Spring Fall Spring Fall Spring Fall Spring Fall Spring 
Year 1 Year 2 Year 3 Year 4 Year 5 
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(a) Enrolled 


Fall Spring Fall Spring Fall Spring 
Year 1 Year 2 Year 3 Year 4 Year 5 


—+— Treatment —-—®-- Control 


(b) Enrolled Four-Year 


Note: Statistics derived from BL data. Year 1 is the academic year beginning in the fall after each high 
school class’ senior year. Treatment line points adds estimated treatment effect from equation (1) to the 
control mean. Sample restricted to cohort 2. 
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Figure A3: Bachelor’s Degree Attainment (6 Years): Heterogeneity, First Cohort 


All Male Hispanic Alt. Prog. Low GPA Low AGI 
Female Black Asian No Alt. Prog HighGPA High AGI 


Sample 
ME Treatment Effect = '———— High/Low 95 Cl 


Note: Statistics derived from BL and NSC data. Treatment estimates provided by estimating equation (1) 
within each subgroup. 


Figure A4: Effects on 5-Year Bachelor’s Degree Attainment: Heterogeneity by High Grad Rate 
PTE 


BA Treatment Effects by PTE of High Grad Rate Enrollment 


1 Treatment Gffect on 5-Yeay BA Attainment 


Below Median Above Median 


(ME Treatment Effect ‘+———— High/Low 95 Cl 


Note: Statistics derived from BL data. PTEs produced using covariates included in main specification. See 
text for additional details. 
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Figure A5: The Relationship Between Treatment Effects on Four-Year Enrollment and 5-Year 
Bachelor’s Degree Receipt 
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Note: Statistics derived from BL data. PTEs produced using covariates included in main specification. See 
text for additional details. We collapse the individual PTEs for bachelor’s degree attainment into means 
within bins of size 0.02. The size of each dot reflects the sample size represented. 
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Table Al: Encouraged Colleges 


College Names Graduation Rate Tuition and Fees Net Price (0-48K) 
Bentley University 84.1 41110 20544 
Boston College 92.2 45622 16196 
Boston University 83.9 44910 23573 
Bridgewater State University 04.4 8053 14680 
Buffalo State SUNY 48.1 7022 8021 
CUNY Hunter College 45.7 6129 5258 
CUNY John Jay College of Criminal Justice 43.1 6059 3993 
CUNY Lehman College 34.9 6108 3297 
CUNY New York City College of Technology 13.6 6069 5220 
CUNY York College 25.6 6096 4590 
Clark University 79.8 39550 18293 
College of the Holy Cross 92.9 44272 15607 
Fitchburg State University 50.8 8985 9013 
Fordham University 81 43577 23352 
Framingham State University 51.5 8080 12515 
MCPHS University 66.4 28470 29807 
Northeastern University 78.5 41686 20140 
SUNY at Albany 64.4 8040 11019 
Saint Joseph’s College-New York 67.5 21878 10292 
Salem State University 45.4 8130 11800 
St Francis College 51.9 20700 9448 
State University of New York at New Paltz 72.7 7083 9844 
Suffolk University 55.9 31716 22900 
The Sage Colleges 51.8 28000 14834 
University of Massachusetts-Amherst 70.4 13258 12437 
University of Massachusetts-Boston 37.9 11966 8084 
University of Massachusetts-Dartmouth 49.9 11681 12581 
University of Massachusetts-Lowell 53.8 12097 10258 
Wentworth Institute of Technology 64 29200 25754 
Worcester Polytechnic Institute 83.5 42778 27224 
Worcester State University 51 8157 10907 
Mean 59.6 20854 13919 


Al 


Table A2: Treatment and Control Assignments 


Boston New York Worcester Total 


Control 193 450 92 735 
Treatment 860 582 245 1,687 
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Table A3: Basic Mediation Analysis for 5-Year Bachelor’s Degree Attainment 


Explanatory Variable (1) 
2-Year College 0.105*** 
(0.036) 
4-Year College 0.427*** 
(0.027) 
High Grad Rate 0.279*** 
(0.020) 
Treatment 0.016 
(0.020) 
Observations 2422 


Note: The table contains the es- 
timats from reestimating our ba- 
sic specification while simultane- 
ously conditioning on enrollment 
and enrollment quality. Each row 
contains the coefficient on an ex- 
planatory variable. As in our ba- 
sic specification, we control for 
site by cohort (i.e., risk set) in- 
dicators. Robust standard er- 
rors in parentheses. * (p<0.10) 
**(D<0.05), ***(p<0.01). 
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Table A4: Advisor Interaction Patterns 


Mean 
Ever Interact with Student (proportion): 0.97 
Office Meeting 0.95 
Phone Meeting 0.32 
Interactions per Student (number): 13.06 
By Medium: 
Office Meeting 8.81 
Phone Meeting 0.42 
Text or Email 0.28 
By Subject: 
First Meeting 2.13 
Second Meeting 1.37 
Application Meeting 3.47 
Financial Aid Meeting 2.03 
Missed Meetings 0.59 


Estimated Contact Time per Student (hours): 10-15 


Note: Statistics calculated from BL data. Sample for 
rows (1)-(3) includes all students assigned to treatment 
and has a sample size of 1687. Remaining rows are 
restricted to the 97.2 percent of students assigned to 
treatment who had any post-assigment interaction with 
BL. Sample size for these rows is 1639. 
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Table A5: Survey Response Balance 


Full Sample Respondents 
Control Mean Control Mean Treatment 
(1) (2) (3) 
Female 0.697 0.755 -0.051 
(0.035) 
Black 0.302 0.335 -0.029 
(0.036) 
Hispanic 0.325 0.335 -0.018 
(0.037) 
Asian 0.246 0.265 0.033 
(0.036) 
Other Race 0.094 0.086 0.009 
(0.023) 
Citizen 0.787 0.767 -0.070** 
(0.034) 
Verified GPA 3.264 3.320 0.018 
(0.043) 
Parent AGI 22520 22295 -720 
(1456) 
Household Size 4.26 4.32 -0.014 
(0.13) 
Mom Employed 0.641 0.532 -0.017 
(0.040) 
Mom Employed (missing) 0.144 0.171 -0.040 
(0.028) 
Dad Employed 0.435 0.352 0.069* 
(0.039) 
Dad Employed (missing) 0.446 0.494 -0.040 
(0.040) 
First Generation 0.811 0.820 -0.048 
(0.032) 
Sibling College 0.389 0.367 0.010 
(0.039) 
Sibling College (missing) 0.059 0.053 -0.002 
(0.017) 
Sibling Bottom Line 0.075 0.053 0.017 
(0.019) 
Sibling Bottom Line (missing) 0.074 0.065 0.009 
(0.021) 
Other Program 0.444 0.412 -0.047 
(0.038) 
Observations 2422 813 


Note: Column (1) contains control group means for the full sample. Column 
(2) contains control group means for survey respondents. Each cell in column 
(3) contains a coefficient from a separate regression of the observed charac- 
teristics on a treatment indicator variable, controlling for site by cohort (i.e., 
risk set) indicators. Response rates did not differ significantly between con- 
trol and treatment groups. Response rate for control group was 0.558, with 
a 0.016 (se 0.029) coefficient on a treatment indicator variable, controlling for 
site by cohort indicators. Robust standard errors in parentheses. * (p<0.10) 
**(p<0.05), ***(p<0.01). 
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Table A6: Effects on Degree Attainment by Site 


Boston Area NYC 


Control Mean ‘Treatment Control Mean ‘Treatment 


(1) (2) (3) (4) 
Enrolled 0.811 0.046* 0.829 0.059*** 
(.025) (0.022) 
Enrolled 4-Year 0.653 0.090*** 0.724 0.089*** 
(.029) (0.026) 
Enrolled High Grad 0.298 0.047* 0.260 0.119*** 
(.028) (0.028) 
BA Degree (5 Years) 0.418 0.074** 0.504 0.072** 
(0.030) (0.031) 
Observations 1390 1032 


Note: Odd columns contains control group means. Each cell in an even column 
contains a coefficient from a separate regression of each outcome variable (indi- 
cated by the row title) on a treatment indicator variable, controlling for site by 
cohort (i.e., risk set) indicators as well as the covariates indicated in Table 1. Ro- 
bust standard errors in parentheses. * (p<0.10) **(p<0.05), ***(p<0.01). 
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Appendix B: Quasi-random Advisor Assignment 


We explore the notion of random assignment of students to advisors more formally by conduct- 
ing a set of randomization tests. In Table B1, we explore the relationship between a number of 
advisor characteristics and baseline student characteristics. Formally, we estimate the following 


specification: 


Ci =at BXit+ er + Ej (2) 
j 


where C; are observable demographic characteristics of the advisors and measures of the extent 
to which a advisor meets with his or her assigned students, and X; includes baseline demographic 
student characteristics. The 1;; are site by cohort fixed effects which control for site by cohort 
variation in the pool of students randomized across advisors. 

The advisor interaction measures (in columns(5) through (8), indicate the average number of 
meetings of each type that a advisor holds over the course of the program. For example, the depen- 
dent variable in column (5) is the average number of meetings about applications that a advisor 
has had with each of his or her students. We follow a leave-one-out procedure to eliminate the pos- 
sibility that a particular student could influence his or her advisor’s score via their own behavior; 
thus, our variable of interest takes the form X_;,,. The estimates in Table B1 suggest little rela- 
tionship between advisor observables characteristics (or behavior) and baseline individual student 
characteristics, supporting the argument that advisors are as good as randomly assigned. F tests 
for the joint significant of all the pre-determined variables are generally insignificant, illustrating 
that particular types of students do not appear to be assigned to particular types of advisors.*? 
Similarly, columns (6)-(9) indicate that particular types of students do not appear to be assigned 
to advisors who exhibit different advising tendencies. This suggests that students are as good as 


randomly assigned to advisors. 


33The lone exception is for white advisors, a result that appears to be driven by white advisors adjusting verified 
GPAs rather than non-random assignment. If we exclude verified GPA from the regression, the remaining variables 
are not predictive of having a white advisor. 
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Table B1: Tests of Random Advisor Assignment 


(1) (2) (3) (4) (5) (6) (7) (8) 
# of # of # of # of 
Couns. Chars. Female Black White Hispanic App. Fin. Aid Office Contacts 
Baseline Covariates: 
Parent AGI -0.000 0.000 -0.000 -0.000 -0.000 -0.000 -0.001 -0.001 
(0.001) (0.001 0.001) 0.001 (0.000) (0.000 (0.001) 0.001) 
Household Size 0.006 -0.002 0.012** 0.001 0.005 -0.001 0.006 -0.002 
(0.007) (0.006 0.006) 0.007 (0.005) (0.005 (0.010) 0.015) 
Verified GPA -0.008 0.014 -0.022 -0.014 -0.021 -0.002 -0.057* = -0.120*** 
(0.020) (0.018 0.018) 0.020 (0.014) (0.013 (0.030) 0.045) 
Female 0.013 -0.006 0.032 -0.030 0.030* 0.018 0.072** 0.087 
(0.024) (0.021 0.021) 0.023 (0.016) (0.016 (0.036) 0.053) 
White or Asian -0.006 -0.037 0.023 0.034 0.030 0.025 0.057 0.108 
(0.042) (0.038 0.038) 0.042 (0.029) (0.028 (0.063) 0.094) 
Black -0.062 0.027 0.015 -0.016 -0.038 0.013 -0.021 -0.014 
(0.041) (0.036 0.036) 0.040 (0.028) (0.026 (0.061) 0.090) 
Hispanic -0.053 0.002 0.007 -0.008 -0.018 -0.006 -0.051 -0.036 
(0.041) (0.037 0.037) 0.041 (0.028) (0.027 (0.062) 0.091) 
Observations 1,596 1,596 1,596 1,596 1,596 1,596 1,596 1,596 
R-squared 0.007 0.008 0.013 0.005 0.010 0.004 0.010 0.009 
Prob>F 0.362 0.262 0.0208 0.702 0.0912 0.728 0.106 0.155 
Mean 0.727 0.228 0.281 0.282 3.591 2.121 9.151 13.58 


Note: Each column contains a regression of a different advisor characteristic on the full set of covariates, 
controlling for site by cohort indicators. The average # of meetings variables are constructed using a leave 
one out procedure, so that each individual is assigned the average number of meetings occurring between 
every other student with the same advisor. Robust standard errors in parentheses. * (p<0.10) **(p<0.05), 


**(p<0.01). 
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